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Preview of Adventures es 





a Autonomous vehicle safety 
e The hard part is perception/prediction 


e Automated identification of perception 
mistakes 





= Vehicle Safety 
e Unintended Acceleration & the pedal misapplication narrative 
e UL 4600: a safety standard for self-driving cars 


Also, personal experiences being an agent of change 
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Washington DC to San Diego 
e CMU Navlab 5 

e Dean Pomerleau 

e Todd Jochem 


https://www.cs.cmu.edu/~tjochem/nhaa/nhaa_home_page.html 


AHS San Diego demo Aug 1997 
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The Big Red Button era 
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APD (Autonomous Platform Demonstrator) 


Safety critical speed limit enforcement 
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Safety Envelope Approach to ML Deployment Mellon 
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= Specify unsafe regions oprt STATE Sp 
‘ ‘ FAILSAFE ACTIVATED C& 

= Specify safe regions en 
e Under-approximate to simplify i ’ UNSAFE! 3 
s 
= Trigger system safety response E OPERATIONAL rt 
upon transition to unsafe region o STATE SPACE = 
Z. / = 
E i 


FAILSAFE ACTIVATED 
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= RSS 
Following 
Distance 
Equation 
e Intel/ 
Mobileye 





—Arnmax,accel 
A ——<— <—V, Amin,brake > 





Figure 1. Reference vehicle geometry for leader/follower. 


This yields a minimum following distance (id., Lemma 2): 


2 2 
! 1 Vrt PAmax,acce . 
d min — MAX 10, (v.p = 2 7 ee |e + ( v = . )t (1) 


2Amin,brake 2Amax,brake 


Where 1n our case the ego vehicle is the following (“rear”) vehicle, and: 
e = d'min 1s the minimum following distance between the two vehicles for RSS 
vr 1s the longitudinal velocity of the lead (“front’’) vehicle 
v; 1s the longitudinal velocity of the following (“rear”) vehicle 
o is the response time delay before the ego (rear) vehicle starts braking 
Amax.brake 1S the maximum braking capability of the front vehicle 
Amax.accel 1S the maximum acceleration of the ego (rear) vehicle 
Amin.brake 1S the minimum braking capability of the ego (rear) vehicle 





: " - ; : EDGE CASE 
Validating an Autonomous Vehicle Pipeline © RESEARCH 
/ Y 
s O 

TRAJECTORY VEHICLE Ly & 
a PERCEPTION PLANNING EXECUTION CONTROL =! 
LU = 
Y) O 
Machine Randomized Control Autonomy << 
Learning & Heuristic Systems Interface To 
Based Algorithms Vehicle 
Approaches = Control 
=> Run-Time Software > Traditional 
> 2??? Safety Envelopes Validation Software 
(~don't hit stuff) Validation 


ML for perception/prediction is uniquely difficult to assure 
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Perception Builds the World Model oe 
snd : 
eo™ 
ve 
ck 

THE REAL 

WORLD 
z 
O COMPUTER’S 
F| | WORLD MODEL: ,_ Path Planning 
Lu “Child chasing & 
O ball into street Motion Control 
n 10 meters ahead’ 
Oo. 
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Free space: available drivable area 


e Move to where the free space is going to be 
e Requires fine grain classification 
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Machine learning uses training data 
e ‘Learns’ via visual features in a picture 














The proverbial 
Black swan 


https://bit.ly/3a2cFL7 
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Good for identifying “easy” cases 
e Expensive and potentially dangerous 
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Unrealistic to Brute Force Safety © RESEARCH 








= If 100M miles/fatal mishap... iii chicas 
e Test 3x—10x longer than mishap rate —— — 
=> Need 1 Billion miles of testing ion 
= That's ~25 round trips 
on every road in the world at 
e With fewer than 10 critical mishaps ~~ 
Y 
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Safer than road testing, but not scalable 
e Simulation is scalable 











Volvo / Motor Trend 
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It’s All About The Edge Cases © RESEARCH 


- - | PREDICTED CONCEPT 


= Gaps in training data can be : | “ 
lead to perception failure 4 
e Safety needs to know: va uN Ez oe 
“Is that a crossing pedestrian?» WF 
e Machine learning provides: 
“Does that look like the 
crossing pedestrians 
in training data?” — 









ond Tas « feather @.970 

"PA ae. * 
F ‘s nature 8.963 
poultry 8.954 
outdoors 8.936 


color @.910 


animal @.908 


https://www.clarifai.com/demo 


= Edge Case are surprises 
e = Edge cases are the stuff you didnt think of! 
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What do we mean by a perception “edge case”? 
Autonomy can’t avoid what it doesn’t see... 
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Customer’s detections 
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EDGE = IN DATASET STROLLER" 














~~ EDGE CASE: 
ae SIGN 


Edge Cases nett variations. 
of everyday objects that are 
-missifg from (training Gata. 


y iy} = a \ PERSON >= =Ie) | 7 





EDGE CASE: STROLLER 


EDGE CASE: 
LOW CONTRAST 
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Automatically Detecting Edge Cases © RESEARCH 


= Adding noise to an image causes objects to drop out 
e Reveals systematic perception issues on unlabeled data 


CUSTOMER 








| PERCEPTION y weer 
o ALGORITHM 
ae ts a Ep Edge Case 
senccerion LI wetecioe 
ALGORITHM if object lists 
differ 





Simplified example: 
add light Gaussian Noise = HOC| OGRAM 
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Pedestrian False Negative = @ kestanch 


EDGE CASE 


Finding Edge Cases Experimentally © RESEARCH 








= Noise randomly perturbs data sample in decision space 
e Change in classification detects points near decision boundary 
e Many of these are Edge Cases (unknown unknowns) 
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“Sun glare” ? 


Baseline, un- waligibenied images with Mask-R CNN 


// Your mileage may vary. © 2020 Philip Koopman 21 
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Many surprises arent obvious to humans 
e Unlikely to be included in human-designed tests 


ia ‘ 








Example: 
High visibility clothing 
missed by perception 


Pilot study on real system: 


e 82% recall of false negatives 
compared to ground truth 


“Single Lane Control” 
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Recalls are the MainNHTSA Hammer tiny 
a Small sampling NHTSA recalls (confirmed bugs) SNE 





17V-713: Engine does not reduce power / ESP software TSA 
17V-686 and MANY others. Airbags disabled NATIONAL HIGHWAY TRAFFIC 
15V-569: Unexpected steering motion / loss of control 

15V-460 and others. Airbags deploy when they should not 

15V-145: Unattended vehicle starts engine> carbon monoxide poisoning 
14V-370: Turns off headlights when driving 

14V-204: 1.5 seconds reverse while displaying Drive 





Voluntary Recalls: 
e 2018 hybrid engine stall at high speeds (https://bloom.bg/2y21T71) 
e 2014 sudden unintended acceleration (https://goo.gl/R9ZgL1) 
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May 25, 2010 


Toyota "U nintended 





tattle ‘Has Killed S93 


ey: 





A 2005 Toyota Prius, which was in an accident, is seen ata police station in Harrison, New York, Wednesday, 
March 10, 2010. The driver of the Toyota Prius told police that the car accelerated on own, then lurched down a 
driveway, across aroad and into a stone wall. (AP Photo/Seth Wenig) AP PHOTO/SETH WENIG 


od Ba 


Unintended acceleration in Toyota vehicles may have been involved in the deaths 
of 89 people over the past decade, upgrading the number of deaths possibly linked 
to the massive recalls, the government said Tuesday. 


The National Highway Traffic Safety Administration said that from 2000 to mid- 
May, it had received more than 6,200 complaints involving sudden acceleration in 
Toyota vehicles. The reports include 89 deaths and 57 injuries over the same 
period. Previously, 52 deaths had been suspected of being connected to the 


problem. 
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http://www.cbsnews.com/news/toyota- 
unintended-acceleration-has-killed-89/ 
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It's All Your Fault: The DOT Renders Its Sth 





Verdict on Toyota's Unintended- a 
Acceleration Scare = US DOT misrepresented 
The final word on the Toyota unintended-acceleration mess. th e Ni AS A repo rt 

CSABA CSERE JUN 9, 2011 = 2012: Class action settles 


= 2013: Jury trial concluded 
it was the electronics 


e ~500 settlements 
= 2014: $1.2B criminal fine 


m= My one hour talk on this: 
https://youtu.be/NCTf7wT5WRO 
(Or search: Koopman Toyota UA ) 








oN 
\ 


| ee Ue 
Sey OM SAER ps ://www.caranddriver.com/features/a1 5125313/its- 
all-your-fault-the-dot-renders-its-verdict-on-toyotas- 
unintended-acceleration-scare-feature/ © 2020 Philip Koopman 25 
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= From the June 2011 issue of Car and Driver 


“It's the Drivers’ Fault” 


However, for most SAI, the most plausible cause of 
an open-throttle condition while attempting to brake is pedal 
misapplication, which is likely to be perceived as brake failure. = 

- Pollard & Sussman, 1989 

= “Most crashes are due to human error, therefore all 
unexplained crashes are due to human driver error’ 
e These statements trace back to this 1989 report 
- Note: the reasoning Is a logical fallacy 
e US DOT reports fail to rule in software as a possible cause 
= Investigations: 
e No mechanical cause found = driver error 
— Compelling facts supporting human results in “unexplained” 
e Non-reproducible behavior = driver error 
— “Pedal Misapplication” often blamed 


https://www.wired.com/2010/03/unintended-acceleration/ 
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2010 





Wea ReaD Operator Erro 


OPERATOR 
ERROR USUALLY 
THE CAUSE OF 
UNINTENDED 
ACCELERATION 


IN PAST 
IWESTIGATIONS 





© 2020 Philip Koopman 26 


Carnegie 


Birth of the Pedal Misapplication Narrative ag taty 


= Audi 5000: before full authority computer throttle control 
e Public narrative: driver pedal mis-application & pedal placement 
2 the same Pollard & Sussman report saying “pedal misapplication!” 


Among the principal conclusions were: 1) Some versions of Audi idle-stabilization 
System were prone to defects which resulted in excessive idle speeds and brief 
unanticipated accelerations of up to 0.3g. These accelerations could not be the 
sole cause of SAIs, but might have triggered some SAIs by startling the driver. 2) 
The pedal and seating arrangements of the Audi are significantly different from 
larger domestic cars. These differences may contribute to a higher incidence of 
pedal misapplication, especially for relatively unfamiliar drivers. 3) Brake 
failures are very unlikely and would be detectable after the event if they 
occurred. 

Pollard & Sussman, 1989, DOT-TSC-NHTSA-88-4 Appendix H; 1983-85 Audi 5000 


Note: 0.3g is 0-to-60mph in 9.1 seconds; 1983 Audi 5000S 0-60 track time is 10.7 sec. 


https://www.zeroto60times.com/vehicle-make/audi-0-60-mph-times/ © 2020 Philip Koopman 27 
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Actual Pedal Misapplication Data ey 


= Gas/Brake confusion 1 out of 997 (Pre-ETC data) 






















— at cee [7] Misapplication of pedals 
Failed to observe 

| 80 _| Weather/adhesion related 

| 60 | Distraction 

Avoiding/hitting obstruction in road 
|  §] ‘| Failure to yield/stop 

Undetermined 






pep =? Foot slipping off brake 
si 507 t gas pedal instead. of brake 
|} [Floor mat wotiged under accelerator 








ae = Other data supports this 


ed eeree aii 
Avoiding vehicle 
Willfl act = Contradicting reports fail to take 
| 45 _ | Driver incapacitation : op cope 
Other into account possibility of 


software defect 
Total: 95 


Figure A27b. Reason/excuses taxonomy. Wierwille at al., FHWA-RD-02-003, 2002 
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https://www.nytimes.com/201 7/ 
10/24/opinion/self-driving-cars- 


LETTERS safety.html 


Will Self-Driving 
Cars Improve Safety? 


Oct. 24, 2017 f vy ‘ed | 





Carnegie 
Mellon 
University 


= Wont drive 


drunk 


= But wont be 
perfect 

= Need safety 
standards 

= Will self- 
certification 
work? 
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The Day Automotive Safety Changed (CQ vesacch 


November 7, 2017 


= If there is no driver... 


e Who do you blame when something 
goes wrong? 


e (Hint: not the driver) 


= Existing safety standards & 
practices are essential... 


e (Car companies should actually 
follow them.) 


e But more is needed for autonomy 





Waymo is first to put fully self-driving 
cars on US roads without a safety driver 
Going Level 4 in Arizona 

By Andrew J. Hawkins | @andyjayhaw 


fo WB omne https://bit.ly/2Vjjvrv 


=\AYMO IS TESTING ITS _ 
SELF-DRIVING MINIVANS 


— SS 


WITHOUT HUMAN DRIVERS 
ON PUBLIC ROADS , ... 


> 
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UL 4600 Autonomous Vehicle Safety CQ fesearct 


= Standardizes what to put in a safety case: 
e Why do you think you are safe? 
e Where is the evidence to prove your reasoning is correct? 


e There are many known hazards: 
#DidYouThinkof That? 


= Underwriters Laboratories / ANSI Standard 
e Non-profit Standards Development Organization 
e Majority of starting point written by Edge Case Research 
e Improvements in response to hundreds of comments 


— Special thanks: 
Uma Ferrell, Frank Fratrik, Deborah Prince, Jason Smith 
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__ System & Lifecycle Issues _ CB researc 

= Drivers do more than drive - — : 
there is no ‘captain of the ship” 
e Ensure ready to operate 


e Mitigate equipment failures 


= Safety related lifecycle participants 
e Inspection & maintenance accuracy 
e Supply chain faults 
e Field modifications & updates 





https://bit.ly/2GvDkUN 


Is it safe to drive now? 


= Safety culture for all stakeholders 
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RESEARCH 


One year / ~ 300 pages 





Mandates safety case approach 
e Why do you think youre safe? 
e What evidence support? 


- 
Ty . 
a 


#DidYouThinkof That? catalog 

e Avoid missed hazards ot — ~ 

e Avoid pitfalls ee 

e Mechanism for industry to share 
hazards & lessons learned 
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Webinars 


UL4600.com 


TECHNICAL WEBINAR GENERAL AUDIENCE WEBINAR 


= Youtube 


ie : e 
preccran & oie each Deborah Prince, Underwriters Laboratories 
F ee See eee : Dr. Philip Koopman, Edge Case Research 
x Jy «, a vr 


GE CASE 
RESEARCH 


UL and Edge Case Research provide a technical UL and Edge Case Research provide an 


(o [ive s-} (0) a=] 0106] MO) a-1010]0 pms) e-]alel-] a0 mie) msy- 11-18) overview of UL 4600: Standard for Safety for the a Down : oad fu ; : 


ice) and a\=m =aV/-} (0-110) pme)m-\elue) ale) anlele cm 4 gelele (al ace saves} 10 r= 180) ame) WA\6lKe)ale)anle)6|om agelel6 (alam halls 


The webinar provides an overview of the presentation is for regulators, policymakers, 
(o folelUaal-Jalar-[arem ale mcom oy-]ad(el| er-]e-mlameal-ma-\U(-\ 1) aat=re|t-Par-] ale ale) anita) aller] B-ie-].<-1 ale) (e(-16-am nal co py of di raft 
process. We} [at-] mudi mel sree ssw ey-[e, -4gellalemmalt-dalirdalecmmile 

with ISO Standards, and other topics. U L 460 0 
Download Slides for the Technical Webinar 


DYo\ vial for-lo my (e(-1-W ce) mh dal -MCT-1al-)e-) WAVE lel (=) ale) 


Webinar 


Techical Webinar Q&A 


General Audience Webinar Q&A 
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UL 4600 Progress € Peer 
= Final comments on a minor revision pending 
e 150+ non-voting stakeholders from 20 countries 
e Voting Standards Technical Panel members include: 
— Autonomy: Uber, Argo, Aurora, Zenuity, Nissan NA, Locomation 
— Components: Infineon, Renesas, Intel/Mobileye 
— Insurance: Liberty Mutual, Munich RE, AXA XL 
— Government: US DOT, CPSC, PennDOT, MITRE, Oak Ridge NL 
— Universities: York, Nanyang, KTH, Waterloo, Beijing 
— Tools: Edge Case Research, ANSYS 
— Others: Center for Auto Safety, Intertek, UL LLC 


= Expect UL 4600 official issue in March 2020 
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Closing Thoughts ae tns 


: Thanks! 
a Mars PA 1 <i 


anne 





= Self-driving cars safer than humans? 
e That sets the bar pretty high! 


e UL 4600 is a first draft of what it will 
take to get there 





ExiT 259A 









= Fundamental shift in safety 
e Perception/prediction novelty 
e No human Captain of the Ship 


e No human driver to blame when things 
go wrong 
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